Update command to install compatible protobuf dependencies #3887
Open
lavakumarrepala wants to merge 4 commits intomainfrom
Open
Update command to install compatible protobuf dependencies #3887lavakumarrepala wants to merge 4 commits intomainfrom
lavakumarrepala wants to merge 4 commits intomainfrom
Conversation
lavakumarrepala
changed the title
lavakumarrepala
changed the title
achauhan-scc
approved these changes
Apr 22, 2026
kingernupur
requested changes
Apr 22, 2026
Member
kingernupur
left a comment
kingernupur
left a comment
There was a problem hiding this comment.
@lavakumarrepala Thanks for the detailed description and analysis which is very helpful to understand the issue and the fix.
While the fix that you are proposing works, I feel that there is some gap in the understanding.
"MLflow relies on legacy protobuf APIs (e.g., FieldDescriptor.label) that are no longer available in newer protobuf implementations."
My takeaway from the statement above is that the mlflow itself isn't compatible with latest protobuf versions, in which case I would expect the mlflow itself to have this constraint in their library, which doesn't seem to be true given the errors that we are observing.
@jayesh-tanna Can you confirm this?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Issue
Training jobs started failing during MLflow logging with:
AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'label'
As a result:
MLflow metrics and params were not logged and mlflow.search_runs() returned incomplete results.
Downstream notebook analysis failed due to missing columns
Root Cause
The curated environment (AzureML-acpt-pytorch-2.8-cuda12.6@latest) picked up a newer protobuf version (v5+), which is incompatible with MLflow’s proto handling.
MLflow relies on legacy protobuf APIs (e.g., FieldDescriptor.label) that are no longer available in newer protobuf implementations.
Fix
Pin protobuf to a compatible version at runtime:
python -m pip install --no-cache-dir --no-deps "protobuf==3.20.3"
Checklist